Multi-paradigm frameworks for scalable intrusion detection

correlation between the clustered alerts' network activity profiles. Using this technique, alerts from multiple IDS, IPS and ADS sensors can be correlated without the need of normalization, the use of an alert ontology or expert rules. Additionally, our approach does not temporally constrain the correlation process, allowing for long-term trend analysis and knowledge discovery. Although our current focus is on network datasets, our approach can be generalized to support data from host-based tools. To evaluate our technique, we implemented a prototype offline correlation system and generated Snort alert correlation results from non-overlapping, independent one-month and six-month datasets. Security analysts at Los Alamos National Laboratory (LANL) evaluated the one-month dataset. The domain experts were asked to compare the correlation of our technique to that of a control group, which performed correlation by grouping alerts by fields in the alert schema. In our test, the experts preferred our technique to the control group, showing that our generalized correlation technique produces results that are better than an algorithm that groups alerts using the alert schema, as typically done in practice. Additionally, the experts showed a preference towards correlation results utilizing network log data in addition to the information in the alert data set. This shows that adding suitable log data yields improved results, an important factor in expanding our prototype in future study to incorporate additional alert databases for correlation. Lastly, we explore the average entropy of alert clusters in relation to the number of clusters generated by the clustering algorithm and use this metric to tune our results. 5.2. Alert Profile Construction To begin the alert correlation process, related network log records, referred to as events, are gathered for each alert in the database. These events are used to construct feature vectors in the second phase of the process. Defining how events are related to a particular alert is challenging and is dependent on domain knowledge and available datasets. This is a critical aspect of the design and implementation of the system. For our prototype, events are gathered from three datasets: Snort alerts, TippingPoint block logs and network connection records from proprietary LANL software known as NetFIead. These three datasets are related in that they monitor network traffic and do so in